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Bessel convolutions on matrix cones 


Margit Rosier 

Abstract 

In this paper we introduce probability-preserving convolution algebras on cones of 
positive semidefinite matrices over one of the division algebras F = M, C or H which 
interpolate the convolution algebras of radial bounded Borel measures on a matrix 
space Mp^q{¥) with p ^ q. Radiality in this context means invariance under the 
action of the unitary group Up{¥) from the left. We obtain a continuous series of 
commutative hypergroups whose characters are given by Bessel functions of matrix 
argument. Our results generalize well-known structures in the rank one case, namely 
the Bessel-Kingman hypergroups on the positive real line, to a higher rank setting. In a 
second part of the paper, we study structures depending only on the matrix spectra. 
Under the mapping r i-^’ spec(r), the convolutions on the underlying matrix cone 
induce a continuous series of hyper group convolutions on a Weyl chamber of type Bq. 

The characters are now Dunkl-type Bessel functions. These convolution algebras on 
the Weyl chamber naturally extend the harmonic analysis for Cartan motion groups 
associated with the Grassmann manifolds U{p,q)/{Up x Uq) over F. 

1. Introduction 

We start with the basic guiding example which corresponds to the rank-one case of our sub¬ 
sequent constructions. For a natural number p ^ 2, consider the set of regular 

bounded Borel measures on MP which are radial, i.e. invariant under orthogonal transforma¬ 
tions. is a commutative Banach algebra with the usual convolution of measures. 

Transferring this structure to M+ = [0, oo) via the mapping x \x\ = (xf 
one obtains a commutative Banach algebra of Borel measures on M+. Calculation in polar 
coordinates shows that its convolution *p is determined on point measures by 

pTT 

6r *p Ssif) = Cp f{\/r'^ — 2rs cos 6) 9 d9, r,s G M+, / G (7(1^+) 

Jo 

with a normalization constant Cp > 0. Now, the above assignment dehnes a probability measure 
5r *p Ss not only for integer p but for all real p > 1, and it extends uniquely to a bilinear 
and weakly continuous convolution on the space Mfe(M_|_) of regular bounded Borel measures 
on M_|_ which is commutative, associative, and probability-preserving l|Kiji. The interesting 
point about this family of convolutions is that analytic properties which are valid for integer 
indices p, due to their origin in radial analysis on M^, remain true for general indices where no 
longer any group structure is present. For example, consider the normalized Bessel functions 
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jaiz) = oFi{a + 1; —z'^/A) with index a = p/2 — 1. If p is a natural number, then 

ja{\x\) = [ X G 

Jsp-^ 

{da denotes the normalized Lebesgue surface measure on the unit sphere and the ho¬ 

momorphism property of the exponential function entails the product formula 

ja{'f')ja{s) = dr *p 6s{ja)- 

This formula, however, extends to arbitrary indices p > 1, see IWl. The space M_|_ together with 
the convolution *p is a prominent example of a commutative hypergroup, called the Bessel- 
Kingman hypergroup ( [BH ] ). A hypergroup is a locally compact Hausdorff space X together 
with a probability preserving convolution of measures on X which generalizes the measure 
algebra of a locally compact group; in particular, one requires a unit element and the existence 
of an involution on X generalizing the group inverse in a suitable way; for details we refer 
to and Section o below. There is a rich harmonic analysis for commutative hypergroups 
extending the analysis on locally compact abelian groups. In particular, there is a Haar measure, 
a dual space, and a Fourier transform satisfying a Plancherel theorem. In our example (M_|_, *p) 
the dual space consists of the Bessel functions 

P 

Ws{r) = ja{rs), s G M+}, a = - - I, 

and the hypergroup Fourier transform is given by a Hankel transform, 

_ 2-p/2 r 

" r( / 9 i / 
r(p/2) 

This extends the fact that the Fourier transform of a radial function F{x) = /(|x|) G L^{MP) 
is again radial and given by a Hankel transform of / with integral index p. 

In the present paper, we generalize the Bessel convolutions described above to a higher rank 
setting, where the space of “radii” is realized as a cone of positive semidefinite matrices. More 
precisely, we construct convolution algebras on such cones which interpolate radial convolution 
algebras on spaces of non-squared matrices. The setting is as follows: For q and a natural 
number p ^ q consider the space Mp^q = Mp^q{¥) of p x q matrices over one of the division 
algebras F = R, C or the quaternions H. It has the structure of a Euclidean vector space with 
scalar product {x\y) = 9Itr(x*7/) where x* = x*. A function (or measure) on Mp^q is called radial 
if it is invariant under the action of the unitary group Up = Up{¥) on Mp^q by left multiplication, 

Up X Mp^q —> Mp^q , {u,x) e-> ux. (l-l) 

Observe that this action is via orthogonal transformations, and that x and y are contained in 
the same C/p-orbit if and only \ix*x = y*y. Thus the space of C/p-orbits is naturally parametrized 
by the cone Hg = ng(F) of positive semidefinite q x g-matrices over F. If g = I and F = R, 
then n coincides with the nonnegative real line R+ and radiality is equivalent to rotational 
invariance in R^. Generalizing the classical case, there is a radial harmonic analysis on Mp^q 
which is based on polar coordinates with liq as radial part and the Stiefel manifold 

Sp,g = {x G Mp^q : x*x = Iq} = Up/Up-q 

as a transversal manifold. In |FTl , this is developed to some extent within the general framework 
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of analysis on symmetric cones. Indeed, the open cone 

rig = {r e Ilq : r strictly positive definite} 

is a symmetric cone within the space Hq = {x G Mg(F) : x = x*} of Hermitian q x q matrices 
over F which carries a natural Euclidean Jordan algebra structure of rank q; see Section [2.II for 
details. 

For each integer q ^ p, we interpret radial analysis on Mp^q in the concise context of a 
commutative “orbit hypergroup” convolution on the cone Ilg which is derived from the orbit 
structure w.r.t. the action of Up on Mp^q. Similar to the rank-one case, the characters of this 
hypergroup, i.e. the multiplicative functions which make up the dual, are obtained by taking 
the means of the characters on Mp^q - usual exponential functions - over the Stiefel manifold. 
This implies that they are given in terms of Bessel functions on the cone Ilg with (half) 
integer index p = pd/2, where d = dim^F. The Bessel function of index is a hypergeometric 
series of the form 

= X] / /l\li ' xeHq] 

here the summation goes over all partitions {A G Nq : Ai ^ ... ^ Agj, {p)\ is a generalized 
Pochhammer symbol, and the Z\ are renormalized versions of the so-called spherical polyno¬ 
mials associated with the underlying cone. They are homogeneous of degree |A| and satisfy 
(trx)^ = Y^\\\=i.Z\{x) for all k G Nq; c.f. Section ITTl The multiplicativity of the Bessel func¬ 
tions {p = pd/2) with respect to the orbit convolution on Ilg expresses itself in a positive 
product formula. Under the technical condition p ^ 2q, this product formula can be written in 
a way which allows analytic continuation with respect to the index p. We thus obtain a positive 
product formula for all Bessel functions with index p ^ d{q—1/2), and an associated continuous 
series of hypergroup structures *p on the cone Ilg whose dual is given by the functions 

{r (ps{r) = s G Bg}. 

Actually, each hypergroup (Ilg, is self-dual in a natural way; the neutral element is 0 and the 
involution is the identity mapping. For matrix cones, the Hankel transform of Elj can now be 
identified with the L^-Fourier transform on the underlying hypergroup; but in addition to the 
results of |FTl, it is also (and primarily) defined as a Fourier transform on an L^-convolution 
algebra. 

Before continuing with structural aspects, let us spend some words on Bessel functions of 
matrix argument. They trace back to ideas of Bochner and the fundamental work of Herz 
and Constantine IHHj in the real case. Much of the interest in these functions is motivated by 
questions in number theory and multivariate statistics. In particular, they occur naturally in 
relation with non-central Wishart distributions, which generalize non-central ^^-distributions 
to the higher rank case, see Eol and |M^ . Nowadays, Bessel functions of matrix argument are 
imbedded into rich theories of multivariable special functions. First, they can be considered the 
oFi class among general pFg-hypergeometric functions of matrix argument, where hypergeo¬ 
metric series are defined in terms of the spherical polynomials; see [GRl] as well as Hi for an 
introduction. Second, all this can be done in the general setting of abstract Jordan algebras and 
symmetric cones, see jFKj . In any case, the spherical polynomials depend only on the eigen¬ 
values of their argument. Considered as functions of the spectra, they can be identified with 
Jack polynomials of a certain index depending on the underlying cone; this was first observed 
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by Macdonald |M2j . There is a natural theory of hyper geometric expansions in terms of Jack 
polynomials (see |Ka.| i which encompasses the theory on symmetric cones. Finally, hypergeo¬ 
metric expansions of such kind are intimately related to the modern theory of hyper geometric 
functions associated with root systems as developed by Heckman, Opdam, Dunkl and others. 
As functions of the spectra, Gaussian hyper geometric functions on a symmetric cone can be 
identified with hypergeometric functions associated with a root system of type BC with a 
specific choice of parameters f [ROj V Similarly, Bessel functions on a symmetric cone can be 
considered as a subclass of the Bessel functions associated with reduced root systems of type 
B in the sense of u which play a fundamental role in the theory of rational Dunkl operators 
EH, m- This connection goes essentially back to jBFj and is made precise in Section 14.Jl 
below. 

In the second part of this paper, we consider structures which depend only on the spectra 
of the matrices from the underlying cone Hg. This amounts to assume invariance under the 
action of the unitary group Ug = Uq{¥) by conjugation, 

r I—> uru~^, u G Ug. 

The orbits under this action are naturally parametrized by the set Hg of possible spectra of 
matrices from Hg, the eigenvalues being ordered by size: 

= {e = (6,..., e,) G K" : 6 ^ ^ ^ 0}. 

Hg is a closed Weyl chamber for the reflection group Bg which acts on M*? by permutations and 
sign changes of the coordinates. Via the canonical mapping from Hg onto Hg which assigns 
to each matrix its spectrum, the continuous series of hypergroup structures (Hg, with jjL ^ 
d{q — 1/2) induces a series of commutative hypergroup structures on the chamber Hg. The 
transfer is established by means of so-called orbital mappings. The convolution, Haar measure 
and dual space of each hypergroup on the chamber are made explicit. In particular, the Haar 
measure of (Hg,o^) is (up to a constant factor) given by where 

v(o=7=Ai - ^(9 -1) -1- 

i=l i<j 

The hypergroup characters turn out to be certain Dunkl-type Bessel functions 

7GHg 

associated with the Hg-root system {±6*, Tejiej} C M'^. Here A: is a parameter on the root sys¬ 
tem which is constant on each subset of roots corresponding to a conjugacy class of reflections; 
in our situation it is given by A: = (ki, k 2 ) with 

d 1 d 

ki = fj. --{q - 1) - - on ±6*; ^2 = - on ± e, ± ej. 

The hypergroup convolution on the Weyl chamber matches the generalized Dunkl translation 
(EH) for Weyl group invariant fnnctions, and we have an interpretation of the Dunkl transform 
as a hypergroup Fourier transform. 

In the geometric cases /r = pd/2, the support of the probability measure <5^ 5g on Hg 
describes the set of possible singular spectra of sums x + y with matrices x,y G Mp^g having 
given singular spectra ^ and y. Further, the characters are just the bounded spherical functions 
of the Euclidean type Riemannian symmetric space {Up x Ug) k Mp^g/{Up x Ug) associated with 
the Grassmann manifold U {p, q) /UpXUg. For general p they are characterized, within the theory 
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of rational Dunkl operators, as the unique analytic solution of a so-called Bessel system; see El. 
It is conjectured that for arbitrary root systems and non-negative multiplicities, the associated 
Dunkl-type Bessel functions satisfy a positive product formula and can be characterized as 
the characters of a commutative hypergroup structure on the underlying Weyl chamber. The 
three continuous series {d = 1,2,4) for Bq obtained in this paper are, to our knowledge, the 
first affirmative examples beyond the group cases associated with Cartan motion groups of 
reductive symmetric spaces. Some background, together with further partial results, is given 
in jR2j. 

The organization of the paper is a follows: In Section 2, background on symmetric cones and 
Bessel functions on cones, as well as some hypergroup analysis are provided. Section 3 is devoted 
to the study of Bessel convolutions on matrix cones. In Subsection 3.1., orbit convolutions 
derived from matrix spaces Mp^q are considered. In 3.3., the corresponding product formula for 
the involved Bessel functions is analytically extended with respect to the index, and in 3.4. the 
associated series of hypergroup convolutions on the cone are studied. In particular, their Haar 
measure and the dual are determined. In 3.5. we analyse an interesting critical index. Section 4 
is devoted to the induced convolution algebras on a Weyl chamber of type B. They are derived 
in 4.1. from the convolutions on the matrix cones and are then, in the two final subsections, 
put into relation to rational Dunkl theory. 


2. Preliminaries 

In this introdnctory chapter we provide some relevant background on symmetric cones, in 
particular matrix cones, and about Bessel functions on such cones. In the main part of the 
paper, we shall introduce orbit convolutions and their “interpolations” on matrix cones within 
the framework of hypergroup theory. As a preparation, a short account on the relevant notions 
and facts is included in the present section. For a general background on hypergroups, the reader 
is referred to the fundamental article (where the notion ’’convo” is being used instead of 
’’hypergroup”), or to the monograph |BHj . An excellent reference for analysis on symmetric 
cones is the book m, for special functions on matrix cones see also [nFF] as well as the 
classical papers of Herz, James and Constantine, 0, E, m 

2.1 Analysis on symmetric cones 

Let F be one of the division algebras F = M, C or the quaternions H. We denote hy t t 
the usual conjugation in F and hy + t) the real part of t € ¥. Consider the set of 

Hermitian q x q-matrices over F, 

Hq = Hq{¥) = {x£ Mq{¥) : X = X*}-, X* = x\ 

We regard Hq as a Euclidean vector space with scalar product {x\y) = fHtr(xy), where tr 
denotes the trace on Mq(¥). The dimension of Hq over M is 

n = q + ^qiq-l), d = dimftF. 

With the above scalar product and the Jordan product x o y = ^{xy + yx), the matrix space 
Hq becomes a Euclidean (equivalently, a formally real) Jordan algebra with unit I = Iq, the 
unit matrix. The rank of Hq, i.e. the number of elements of each Jordan frame in Hq, is q. 

The set flq = Dq(F) of positive definite matrices from Hq is a symmetric cone. Recall that a 
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symmetric cone 0 is a proper, non-empty convex cone in a finite-dimensional Euclidean vector 
space which is self-dual and homogeneous in the sense that its group of linear automorphisms 
acts transitively. Let G denote the connected component of this group. Then K = GCi 0{V) is 
a maximal compact subgroup of G and = GjK, a Riemannian symmetric space. The matrix 
cones are realized as Rq(F) = GLq{¥)/Uq{¥). Hereby GLq{¥) acts via r grg*, which reduces 
to conjugation when restricted to the unitary subgroup Our main interest will be in the 

closure of Og relative to Hq which coincides with the set of positive semidefinite matrices over 
F, 

Hg = ng(F) = {x*x ■. X e Hq} = {x^ ■. X e Hq). 

For a general Euclidean Jordan algebra V, the interior R of the set : x £ R} is a symmetric 
cone, and each symmetric cone in a finite-dimensional Euclidean vector space V can be realized 
in such a way. For details see Section III.3 of m- The simple Euclidean Jordan algebras 
correspond to the irreducible symmetric cones and are classified. Up to isomorphism, there are 
the above series Hq(¥) with F = M, C,]H1, the exceptional Jordan algebra Ll3(0), as well as one 
infinite series of rank-2 algebras corresponding to the Lorentz cones 

An = {{x,Xn) £ M X ; X^ — |x'P > 0, Xn > 0}. 

Let U be a simple Euclidean Jordan algebra of rank q. We recall that for each x £ U there 
is a Jordan frame {ei,..., Cg} such that x = with real numbers Up to ordering, 

the are uniquely determined and are called the eigenvalues of x. Notice that x £ U iff all its 
eigenvalues are positive. The trace and determinant of x are defined by 

Q g 

trx = ^^i, A(x) = ]J^i. 
i=l i=l 

In the Jordan algebras iLg(F), the function A coincides with the usual determinant det if 
F = M or C while for F = H we have A(x) = (detx)^/^ when x is being considered as a 
complex 2q x 2g-matrix in the standard way. 

The basic functions for the harmonic analysis on a symmetric cone, and at the same time 
the basic constituents for all kinds of hyper geometric functions on it, are the so-called spherical 
polynomials. To write them down, we have to introduce some more notation; 

Consider a fixed Jordan frame {ei,...,eg} of V. For 1 ^ j ^ q and x £ U we denote 
by Aj(x) the principal minors of A(x) with respect to this frame; in particular Ag = A and 
each Aj is a polynomial function on V which is positive on the cone fi. Recall that a g-tuple 
A = (Ai,..., Ag) £ Nq is called a partition if Ai ^ A 2 ^ ... ^ Ag ^ 0. The weight |A| of A is 
defined as |A| = Ai Ag. Following a standard convention, we write A ^ 0 in order to 

indicate that A is a partition. For a partition A, the power function A;, on V is defined by 

Aa(x) = • ... • Ag(x)^G 

It is a homogeneous polynomial of degree |A| and positive on U. In particular, if x = Yli=i 
with ,..., ^g £ M then Ag (x) = •... • and A^ (x) = •... • ^g . The spherical polynomials 

of U are indexed by partitions and defined by 

<I>a(x) = / Ax{kx)dk, A ^ 0, x £ U 
Jk 

where dk is the normalized Haar measure on K. Notice that d>A(e) = 1. is iL-invariant 
and therefore depends only on the eigenvalues of its argument. The <I>a are just the polyno¬ 
mial spherical functions of U = GjK. For the matrix cones over M they are known as zonal 
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polynomials, for those over C they coincide with the Schur polynomials. Actually, for each 
symmetric cone the spherical polynomials are given in terms of Jack polynomials l |Stfi,| l. This 
has been first observed by Macdonald |M2j (see also [F] and the notes in |FKj . Chapt. XI). 
More precisely, let us consider the renormalized polynomials 

Z,(x) := dA^-<I>A(x) 

Vg/A 

where dx denotes the dimension of the vector space of polynomials on which is generated 
by the elements z i—> Ax{g~^z), g G G. (c.f. Sect.XI.5. of m)- Let further C" denote the Jack 
polynomials of index a > 0 l |Staj L normalized such that 

(6 + ... + eg)"= cno VfeeNo (2.1) 

A^O, |A|=fe 

(c.f. |Kaj ). Then for x G 1^ with eigenvalues ^ = (^i,..., .^g) we have 


Zx{x) = C^iO 

where Oi = ^ with d = dirngF for the matrix cones over F and d = n — 2 for the Lorentz cone 
An- 

In normalization constants, we shall also need the gamma function of a symmetric cone. 


rf^(^)= f e-*™A(x)"-^/^dx 

Jn 

which converges absolutely for z G C with yiz >|(q — 1) = ^ — 1 and can be written in terms 
of the classical gamma function as 


see |FKj . Chapt. VII.1. 


i=i 


2.2 Bessel functions on a symmetric cone 

Hypergeometric expansions in terms of spherical polynomials have a long history in multivariate 
statistics, tracing back to the work of Herz 0 , James |Jaj and Constantine |Coj . They are 
important in the study of Wishart distributions and for questions related with total positivity 
1 |CR2) 1. In view of the above connection between spherical polynomials and Jack polynomials 
it is natural to treat these classes of functions in the more general framework of multivariable 
hypergeometric functions based on Jack polynomial expansions (see |Kfij L In our context, only 
hypergeometric functions of type qFi (which are Bessel functions) will be relevant. 

Let a > 0 be a fixed parameter. For partitions A = (Ai,... Xg) we introduce the generalized 
Pochhammer symbol 

(c)a =!!(''- A- 

i=i 

For an index g G C satisfying (/r)" 7 ^ 0 for all A ^ 0 (in particular for g with 2fig > ^{q — 1)), 
the generalized hypergeometric function QF^{g] .) on is defined by 


7 






Margit Rosler 


It is known ([K^) that this series converges absolutely for all ^ G C^. Similarly, a oFi- 
hypergeometric function of two arguments is defined by 

1 c^ioc^v) 




A>0 






1 = 


Now suppose that a = 2!d where d is the dimension constant of a simple Euclidean Jordan 
algebra V of rank q corresponding to the symmetric cone R. Then the Bessel function o-E“(/r; •) 
essentially coincides with the Bessel function associated with R in the sense of m- Indeed, 
the latter is defined by 


Thus for a: G R with eigenvalues ^ = (■^i, • • • we have 

Mx) = ofY id-,-0- 

In Section EH we shall also work with Bessel functions of two arguments x,y G R, 

(- 1 ) 1^1 Z^{x)Zx{y) 

,,o(^)^|A|! Z.(e) 


( 2 . 2 ) 


(2.3) 


■= Y 


where e is the unit of R. For G R with eigenvalues ^ = (^i, 
respectively, we thus have 

Jtiix,y) = QFY{ix]ii,ir]). 

The following estimate is especially useful for large indices y.. 


,^q) and rj = (r/i. 


■,Vq) 

(2.4) 


Lemma 2.1. For a > 0 let QFf‘{y, .) denote the associated generalized Bessel function of index 
y on C*^. Suppose that fRy > -^{q — 1). Then for ^ ■ ■ ■ ,^q) ^ 

Proof. For n G Nq and z G C with IHz > 0 we have 


r{z) 


T{z + n) 




T{9\z) 


^ 1 . 


T{y{z + n) 

Therefore |(/^)“| ^ |(91/x)"| ^ 1. Moreover, it is known that the coefficients of the Jack 
polynomial Cf in its monomial expansion are all non-negative (see |KnSj L This implies that 

Using relation (EU), we therefore obtain 


A>0 


□ 


Corollary 2.2. Let denote the Bessel function of index y associated with the symmetric 
cone R inside the Jordan algebra V of rank q. Suppose that TKy > |(g — 1). Then for x G R 
with eigenvalues fi, ■ ■ ■ ^q, 

\Mx)\ ^ 


8 










Bessel convolutions on matrix cones 


We mention at this point that for iH/U > d{q — 1) + 1 and a E O, the Bessel function on 
has the absolutely convergent integral representation (an inverse Laplace integral) 

Mr) = ^ f (r e S}) 

Ja+iV 

see Prop. XV.2.2. of m- This integral was originally used by Herz [Hj to define Bessel functions 
on the cones of positive definite matrices over R; see also [FT] and [Dij for arbitrary symmetric 
cones. Corollary 12.21 slightly improves the order estimate for given in [^, p.486 for F = R. 

2.3 Hypergroups 

We start with some notation: For a locally compact Hausdorff space X, let Mi,{X) denote 
the Banach space of all bounded regular (complex) Borel measures on X with total variation 
norm, and M^[X) C Mb{X) the set of all probability measures. With 6x we denote the point 
measure in x E X. We use the notions C{X),Cb{X) and Cc{X) for the spaces of continuous 
complex-valued functions on X, those which are bounded, and those having compact support 
respectively. Further, C'o(X) is the set of functions from C{X) which vanish at infinity. 

Definition 2.3. A hypergroup (X, *) is a locally compact Hausdorff space X with a bilinear 
and associative convolution * on Mf,(X) with the following properties: 

(1) The map (/x, u) i—> ^ * u is weakly continuous, i.e. w.r.t. the topology induced by C'b(X). 

(2) For all x,y G X, the product 6x*Sy of point measures is a compactly supported probability 
measure on X. 

(3) The mapping (x,y) i—> supp(53; * 6y) from X x X into the space of nonempty compact 
subsets of X is continuous with respect to the Michael topology (see |J^L 

(4) There is a neutral element e E X, satisfying 6e * Sx = Sx * Se = Sx for all x E X. 

(5) There is a continuous involutive automorphism x x on X such that Sx*Sy = {6y * 6x)~ 
and X = y e E supp((5a, * 5y). (For y E Mb{X), the measure is given by y~{A) = 
/x(A) for Borel sets A C X). 

A hypergroup (X, *) is called commutative if its convolution is commutative. 

For a hypergroup (X,*), the space {Mb{X),*) is a Banach algebra with unit 5e- Notice 
that by the density of the finitely supported measures in Mb(X), the convolution is uniquely 
determined as soon as it is given for point measures. 

Of course every locally compact group is a hypergroup with the usual group convolution. 
In our context, only commutative hypergroups will be relevant. Of particular interest will be 
the following 

Example 2.4. ([Ji], Chapt. 8) Let (G, +) be a locally compact abelian group and K a compact 
subgroup of Aut G. Then the space = {K.x : x E G} of X-orbits in G is a locally compact 
Hausdorff space with the quotient topology and becomes a commutative hypergroup with the 
(natural) definition 

{dK.x*5K.y){f)= [ f{K.ix + ky))dk, /EGe(G^). 

Jk 

(G^,*) is called a orbit hypergroup; its neutral element is K.O = 0 and the involution is 
(K.x)- = K.{-x). 
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In the following, we collect some further ingredients underlying the harmonic analysis on a 
commutative hypergroup X. There exists (up to normalization) a unique Haar measure u on 
X, i.e. a positive Radon measure satisfying 


lx 


f{x*y)duj{y) = [ f{y)duj{y) V x G X, / e C'c(X) 

Jx 


where we use the notation 


f{x*y) := / fd{6x*6y). 


IX 


Similar to the dual of a locally compact abelian group, one defines the dual space of X as 


X := {(f e Cb{X) : (p ^0, if{x) = ip{x) and ip{x * y) = (p{x)(p{y) Vx, y G X}. 


This is a locally compact Hausdorff space with the topology of compact-uniform convergence. 
The elements of X are also called the characters of X. They are the constituting functions for 
harmonic analysis on {X, *). The Fourier transform on Mb{X) is defined by 

p{lp) := / ip{x) dy(x), p^X, 

Jx 

and on L}{X,uj) by / := fuj. The Fourier transform is injective, and there exists a unique 
positive Radon measure tt on X, called the Plancherel measure of {X,*), such that f ^ f 
extends to an isometric isomorphism from L?‘{X,uj) onto LP‘{X,'k). While the Haar measure oj 
has full support, the support of vr may be a proper subset of X only. 


Example 2.5. [Continuation of example 12 .4L ] For a Haar measure m on the group G, the image 
measure of m under the the canonical map vr : G —> provides a Haar measure on the orbit 

hypergroup G^. Further, it is easily seen that the functions 

Pa{K.x) ;= / a{k.x)dk, a G G, 

Jk 

belong to the dual of G^. Actually, we have 
Lemma 2.6. 1. G^ = {pa '■ a G G}. 

2. Pa = Pa' iff Oi and a' are contained in the same orbit under the dual action of K on G, 
given by {k.a){x) = a{k~^.x) 

Proof. It is known that G^ can be identified with the extremal points of the set 

H = {/3 G G(G) : (5 positive definite and A-invariant with /3(0) = 1}; 

the identification being given hy (3 pp, pp{K.x) = /3(x) for x G G, see |Hoj . By Bochner’s 
theorem, each /? G H is of the form /3(x) = Jq a{x)dfi{a) with some A-invariant probability 
measure p on G. In this way, the extremal points of H correspond to the measures of the form 
Pa = jx ^k.adk with oi ^ G. Further, pa = Pa' iff « and a' belong to the same A-orbit in 
G. □ 


A commutative hypergroup (A, *) is called self-dual if there exists a homeomorphism T : 
X ^ X such that 

^{x){z)'^{y){z) = [ ^{w){z)d{S^*6y){w) \/x,y,zeX. 

Jx 
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In that case, X is a dual hypergroup in the sense of [Jil, 12.4. where the convolution product 
'^'i'(a;) * ^^{y) is just the image measure of 6x * 6y under 'h. It is easily checked that the image 
measure 'I'(a;) of the Haar measure a; on X is a Haar measure on X. Thus by Theorem 12.4A 
of [Jej, T(a;) coincides up to a multiplicative constant with the Plancherel measure n oi X. 


3. Convolution structures associated with Bessel functions on a matrix cone 


3.1 Orbit hypergroups on a matrix cone 

For natural numbers p ^ q, consider the matrix space Mp^q = Mp^g{¥) of p x g-matrices 
over F. We regard Mp^q as a real vector space, equipped with the Euclidean scalar product 
{x\y) := 91tr(x*i/) and norm ||x|| = ^tr(x*x). Here Tit = + t) is the real part of f G F and 

tr denotes the trace in Mq{¥). In the square case p = q, ||.|| is just the Frobenius norm. Let us 
consider the action of the unitary group Up on Mp^q by left multiplication. 

Up X Mp^q —> Mp^q , {u, x) l-H’ UX. 

The orbit space Mp^ for this action can be identified with the space H^ = nq(F) of positive 
semidefinite g x g matrices over F via 

Up.x ^ Vx*x =: |x|. 

Here for r G Hg , ^/r denotes the unique positive semidefinite square root of r. It is easy to see 
that the above bijection becomes a homeomorphism when Mp^q is equipped with the quotient 
topology and Hg with the subspace topology induced from Mq. (Indeed, both the canonical 
map X i-H’ Up.x from Mp^q onto Mp^q and the mapping x ^/x*x, Mp^q —> Hg are open and 
continuous.) Notice that the Stiefel manifold 


Sp,g = {x G Mp^q : x*X = Ig} 


is the orbit of the block matrix 

o'o ^ Q ^ ^ 

Before calculating the orbit hypergroup convolution for the above action, let us recall the 
basic aspects of radial analysis in Mp^q as developed in Ell. As in the introduction, radiality 
here means invariance under the above action of Up; a function F on Mp^q is radial iff it is of 
the form F{x) = f{\x\) for some / : Hg —> C. Suitable polar coordinates in Mp^q are defined as 
follows: Let da denote the C/p-invariant measure on Sp^g, normalized according to fj. da = 1, 
and let 


M'^q = {xG Mp,g : A(x*x) 0} 
which is open and dense in Mp^g. Then the mapping 

Hg X Sp,g ^ Mp q , (r, a) a^/r 

is a diffeomorphism, and for integrable functions / : Mp^g —> C one has 


with 



f{x)dx 



f{ay/r)X{rydr da 


gj-dpg/2 

fLM 


dp n 

and 7 = - -. 

2 q 


(3.1) 
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As before, n is the real dimension of Hq. Our notion slightly differs from that of Elj (and 
the monograph |FKj i: it is adapted to the left action CU) of Up on Mp q while the notion of 
El! would require to have Up acting from the right on Mq^p. As Lebesgue measure on Mp^q 
is [/p-invariant, the Fourier transform of a radial function in L^(Mp^q) is again radial, and 
also the convolution of two radial functions is radial; both can be calculated by use of polar 
coordinates, c.f. EH- We shall come back to this shortly. Following Example I2.4L we obtain 
the orbit hypergroup convolution 


iSr*ds)if)= f{\aor + uaos\)du, r, sGlIg. 


iu„ 


The image measure of du under the mapping Up —> , u i—> uao is [/p-invariant and therefore 

coincides with the Rp-invariant measure da. Hence, 

iSr*6s)if)= f f{\aor + as\)da = ( f{'\/r‘^ + s^ + rds + {sdr)*)da (3.2) 

J Stj.O Sxj.O 


where d = a^a is the q x g-matrix whose rows are given by the first q rows of a. Actually, this 
convolution depends on p (and q), which we surpress for the moment. The neutral element of 
the orbit hypergroup (Hg, *) is 0, and the involution is the identity mapping (because x G Mp^q 
and —X are in the same t/p-orbit). Further, according to Lemma l2.bl the dual space of (Hg,*) 
consists of the functions tps, s G Hg with 


^s{r) 


-i{uaor\aos) 


'Ur, 





(3.3) 


These are Bessel functions. Indeed, according to Propos. XVI.2.3. of m we have for x G Mp^q 
the identity 

f = p = ^ (3.4) 

where J7)t is the Bessel function of index p associated with the symmetric cone fig as in Section 
O Thus 

>^s{r) = Jp{^rs^r). 

We mention at this point that up to a constant factor, (^^(r) coincides with the Bessel function 
J(r^, s^) in |FT| . As Jp{x) depends only on the eigenvalues of x and as the matrices rs^r and 
sr‘^s have the same eigenvalues, we see that 


ipsir) = tpr{s) Vr, sGHg. 

This implies that the hypergroup (Hg, *) is self-dual via r i—> ipr- In order to stress the depen¬ 
dence of its convolution on p (or equivalently, on the index p = pdj2), we denote it by from 
now on and write Hg^^ for the orbit hypergroup (Hg, *^). 

A Haar measure ujp on the hypergroup Hg^^ is obtained by taking the image measure of the 
(normalized) Lebesgue measure dx on Mp^q under the mapping x e-> |x|. Using again 

polar coordinates, we obtain 

2-m f 

= /(^/f)A(r)^dr, /GC,(ng). 

rn,(/i) Jn, 


Fourier transform and convolution of radial functions on Mp^q are calculated in our notion 
a follows; Suppose F,G G V-{Mp^q) are radial with F{x) = /(|x|) and G{x) = ( 7 (|x|). Then the 
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Fourier transform of F is 


The convolution of F and G is given by T * G{x) = F[{\x\) with 
H{r) = f F{aor - y)G{y)dy 

J Mn.a 


= /(|A|). 


2-m 


f{\aor - cr\/s 1)^ G{^/s)A{syds 


[ Sr*iidsif)gis)dujf,{s) = if *f,g){r). 
Jlla 


The multiplicativity of the characters (ps implies a positive product formula for the Bessel 
functions with index g = pd/2, p q an integer; 

= f + rds + sd*r)da Vr, sellg. (3.5) 

^p,q 

We shall generalize this formula to Bessel functions of arbitrary index /x E C with 9^/i > 
d{q — 1/2). For real indices we shall obtain a positive product formula, which leads to a con¬ 
tinuous family of hypergroup structures on Fig beyond those which have a realization as orbit 
hypergroups as above. The decisive observation towards this aim is that only the reduced ma¬ 
trix a occurs in the integrands of (EH) and (TOl) . In the following section, we shall introduce 
coordinates on the Stiefel manifold which are adapted to this situation. 


3.2 Split coordinates on the Stiefel manifold 

Let fc G N with p — k ^ q. We decompose a G as cr = with v G and w G Mp_k^q. 
For fixed q, put 

Dk = {v e Mk^q : v*v < 1} 

where the notion x < y for x,y G Mq(¥) means that y — x is (strictly) positive-dehnite. 


Proposition 3.1. The mapping 


^ : Dk X 


Tp- 


k,g 




{V,a 


V 

a'VT^ 


is a diffeomorphism onto a dense and open subset Tp^q of Tp^q. Let da and da' denote the 
normalized Riemannian volume elements on Tp^q and Tp-k,q respectively, and let g := ^{p — 
k) — Then on Tp^q, 

da = • A(I — i;*u)^ da'dv 


where 





A{I — v*vPdv) ^ 


^-dkq/2 


rn.(f) 

rn,{^)' 


In the important special case p ^ 2q,k = q with F = M, this result goes back to pj. For 
the reader’s convenience we nevertheless supply a concise proof along a different approach. 
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Proof. It is easily checked that : Dk x 'Pp-k,q ^p,q is and injective, and that its image 
T,p^q is dense in P^p^q- Let / G C(Sp^g) and extend it to F G L^{Mp^q) by F{a^) := /(cr) if 
< r < and F = 0 else. Then 


f F{x)d: 

J Mr),Q 


x = C 


fda with C = C, 


p,g 


L 


A{r)"^dr. 


(3.6) 


I<r<p 


On the other hand, we write x G Mp^q in block form as a: = with xi G M^^q and 

X 2 G Mp_k^q. Then 

[ F{x)dx = Cp_k,q [ II A{r 2 )'^ dr 2 da 2 dxi 

where we used polar coordinates X 2 = (y‘i\/p 2 in ^y-k,q- Consider now the successive transforms 
r 2 e-> x\xi + r 2 =: r with xi fixed and xi^ v := xir~^/‘^ with r fixed. We have dr 2 = dr and, 
by Lemma 2 of EH, dxi = A{r)^^/‘^dv. Thus in a first step, 

I F{x)dx = Cp-kq I I I f( ^ j ^ A(r - dr(icr2dxi 

Jm,,, \(T2 ^r-x^xiJ 


where F 


( —T— I = f ( ^ I with w = \/r — ^v*V\fr ■ r As w*w = I — 

\(y2^r-x\xi) \cr2wj 


V V, 


it follows that w = uy/I — v*v with some u ^ Uq. The invariance of the Lebesgue measure on 
Mp-k,q under the action of Uq by right multiplication easily implies that da 2 is invariant under 
this action of Uq on Tip_k^q. In view of the identity r/ + kd/2 = 7 , the above integral therefore 
becomes 




p—k^q 


JDk 




, L L.. ^ 

7 L L...' 


Together with (EH), this gives the stated Jacobian (which does not vanish on Dk x T,p_k^q) 
and proves the claimed diffeomorphism property of <I>. □ 

If p ^ 2 q, then the above proposition with k = q leads to a nice integration formula for 
functions /(u) on which depend only on the first q rows of cr. For abbreviation, we put 


e ■■= d{q - ^) + 1 


and for /x G C with IH/x > p — 1, 


Kn ■ = 


'Da 


A(I — v*v)'^ ^dv. 


The explicit value of Kp is obtained by using polar coordinates and Thm. VIL1.7. of m about 
beta integrals on symmetric cones, 

,V 2 


Ku = 


Cq,q I A(/ - r)^-^A(r)'^/2-^dr = . 

Jr<I 


“ ro,(^) 

Corollary 3.2. Let p ^ 2q. Then for f G CiTp^q) of the form 

= d = ola 
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one has 


fda = 


1 


f{v) A(/ - 


- ^p,q l^pd/2 JDq 

The dependence on p now occurs only in the density, not in the domain of integration. 


3.3 A product formula for Bessel functions with continuous index 

Remember that the integrand in the convolution formula (TO) and the product formula H3.5I) 
depend only on the reduced matrix a = cr^a. By Corollary 13.21 we obtain 

Proposition 3.3. Suppose that 2q and let p, = pd/2. 

(1) The convolution H3.2|) (i.e. *^) on Ilq can be written as 

*fi ds)if) = — [ f{V + rvs + sv*r ) A{I — v*vY~^ dv. 

JDq 

(2) The Bessel function satisfies the product formula 

+ s'^ + rvs + sv*r) A{I - v*v)^~^dv Vr, sGlIq. 

JDq 

We are now going to extend the integral formulas of Proposition 13.31 to arbitrary indices p 
within the half plane {/r G C : Tip > g—1 = d[q — ^)}. We will use the standard technique (c.f. 
|Ste| for the rank one case), namely analytic continuation with respect to p. The argumentation 
will be based on a classical theorem of Carlson: 

Theorem 3.4 m, P.186. Let f{z) be holomorphic in a neighbourhood of {z £ C : Tiz ^ 0} 
satisfying f{z) = on Tiz ^ 0 for some c < tt. If f{z) = 0 for all z G No, then f is 

identically zero. 

The following theorem is the main result of this section. 

Theorem 3.5. Let p £ C with Tip > g — 1 = d(^q — i). Then the Bessel function satisfies 
the product formula 

+ rvs + sv*r) A{I - v*v)^~^dv Vr, sGlIg. (3.7) 

JDq 

Remark. It is easy to obtain from (EIIl) the more general identity 

Jp[x*x)j^[y*y) = — [ Jp.{x*x + y*y + x*vy + y*v*x)A{I-v*vY~^dv (3.8) 

JDq 

for arbitrary x,y £ Mq. For this, recall that every x £ Mq has a polar decomposition x = u\x\ 
with |x| = y/x*x £ Ilq and a unitary matrix u £ Uq. Choose r = \x\ = u*x and s = |y| = w*y 
with u,w £ Uq in (13.7|) . Then 

jf,{x*x)j^{y*y) = — [ Jp.{x*x + y*y + x*uvw*y + y*wv*u*x)A{I - v*v)^~^dv. 

JDq 

Under the coordinate transform v uvw* =: v we have dv = dv, A{I — v*v) = A(I — v*v} 
and V £ Dq yy V £ Dq. This implies (EH)- 
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Proof of Theorem \,y.5[ Let VL := {/i G C : fH// > — 1}. By the asymptotic properties of the 

usual gamma function we have 


(-) 


dq^/2 


uniformly in W as /r —> oo. 


(3.9) 


Consider now the claimed product formula (EZl). Its left-hand side is holomorphic and, accord¬ 
ing to Corollary 12.21 also uniformly bounded in VL as a function of /U. In order to estimate the 
right-hand side of (Umi . note that the argument -|- rvs + sv*r is positive semidehnite 

for all V G Dq. Moreover, 

tr(r^ -I- -|- rvs -|- sv*r) = ||r|p -|- ||s|p -|- 2{r\vs) 

^ ||r||^-|-2||r|| ||us||-|-||s|p ^ (Ikll + llsH)^ (3.10) 


where again || . || is the Frobenius norm on Mq. For the last estimate, it was used that I — v*v 
is positive-definite and therefore 


||s||^ — ||us|p = tr(s(/ — v*v)s) ^ 0. 

Thus by Corollary 12.21 

+ rvs + sv*r)\ < \/v e Dq. 

This easily implies that the right-hand side in (EH) is also holomorphic as a function of // in 
W, and can be estimated according to 

— / J'^[r'^ + s"^ + rvs + sv*r) A{I — v*v)^~^dv 

JDq 

A{I-v*v)^>^-^dv = C ■ (3.11) 

l^/i| JDq 

with a constant C > 0 independent of In view of the last expression is of the form 

odMi"’"'") as /X ^ oo in W. 

Now define f{z) := g[{z + 2g)|) where 

g{n) := J^{r‘^)J^{s^) - — [ J'^(r^ -h rvs + sv*r) A(/ - v*vY~^dv, g€W. 

JDq 

Then f(z) = 0 for all 2 ; G Nq by Proposition 13.31 The above considerations further show that 
/ is holomorphic and of polynomial growth on Dlz > — 1. With Carlson’s theorem, validity of 
(EH) follows as claimed. □ 


Remarks. 1. We may as well establish a Bochner type integral representation for the Bessel 
function with > g—lhy analytic continuation. Indeed, for /x = ^ with an integer p q, 
we obtain from E31) that for all x G Mq , 




{x*x) = [ 

Jt., 






If p ^ 2g then according to Corollary 13.21 this can be written as 


Mx*x)= — [ 

JDq 


(3.12) 


Analytic continuation with respect to g as above shows that (IXTl) remains valid for all 
|U G C with 93/x > g—l. From this identity, it follows by the Riemann-Lebesgue Lemma for the 
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additive matrix group (Mq,+) that G C'o(ng). In case F = M, the integral representation 
(ITT^ for > Q — 1 goes back to j^, where it was proven by a different method. 

2. In the rank one case g = 1 we have 

2 

with the one variable Bessel functions 


Ja{Z) = 


TP ( _l_ 1 \ (-l)^r(a + 1) iZ-^2n 

0 n"+ ’ a) ^n!r(a + n + l)*' 2 ^ 


n=0 


Formulas (IXT^ and (EZZl) reduce to the well-known Mehler formula 


ja{r) = caj 


and the product formula 


r 1 

ja{r)ja{s) = Ca J ja {V+ 2rst ) (1 - 

which are both valid for a G C with 9\a > —1/2. 

We finish this section with an alternative form of product formula (EH) and the integral 
representation (IXni) for the Bessel functions Let 

B := {z G Mi^q{¥) : \z\ < 1} 

denote the unit ball in Mi^g(F) = F^ with respect to the standard norm \z\ = ■ 

It is easily checked that 

Lemma 3.6. The mapping 


A{I-z*z) = 1-1^1' 


P{yi,---,yq) ■= 


yi 

2/2(L-yJyi)^/2 


; yi,...,yq£ B 


\yqiI-y;_,yq-i)PP--iI-ylyi)Py 
establishes a diffeomorphism from B'^ onto Dq with the Jacobi determinant 


9-1 


detdP(y)| = 

i=i 


Proof. For k with 2 ^ k ^ q, the mapping 

B X Dk-i —> Dk, {z,w)^( ^ 

\w\J I — z*z 

is obviously a diffeomorphism. By Lemma 2 of Ell and Fubini’s theorem, its Jacobi deter¬ 
minant is given by (1 — . {q — l)-fold iteration of this decomposition yields the 

assertion. □ 
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It is easily checked that for v = P{y) G Dg, 

A{I-v*v) = f[AiI-y*y,) = ]J (1 - ). 

j=i i=i 


This implies 

Corollary 3.7. Let C G c with > “1- Then the image measure of A{I — v*v)‘^dv under 
P~^ : Dq^ B‘i is given by nj=i(l “ 


Application to the product formula (EZI) and the integral representation for the Bessel 

function gives 


Corollary 3.8. Let /r G C with tRy , > g — 1 . Then for all r,s € Liq , 

+ s‘^ + rP{y)s + sP{y)*r) ]T(1 - 
I^U JB<! 1 

MP{y)\r) _ |y^.|2)M-e+%-L/2^y. 

i=i 



3.4 Bessel convolutions on the cone Ilg 

For real y > g — 1 the measure ■ A(I — v*v)^~^ in product formula (I3.7jl is a probability 
measure on Dq. We shall see that (EH) leads to a hypergroup convolution on 11 ^ which is of 
the same form as those of Proposition 13.3L This will give us a continuous series of commutative 
hypergroup structures on Ilg which interpolate those occuring as orbit hypergroups for the 
indices y = pd/2, p 2q an integer. 

Theorem 3.9. Fix an index y G M with y > g — 1. 

(a) The assignment 

(^dr Ss){f) '■= — [ /(\/r 2 + s 2 _|_ j.yg _|_ _ v*vY~^ dv] fGCiJlq) 

J Dq 

defines a commutative hypergroup structure = (Ilg , *^) with neutral element 0 and 
the identity mapping as involution. The support of Sr Sg satisfies 

supp{Sr *IJ, Sg) c {t eHq : ||t|| ^ ||r|| + ||s||}. 

(b) A Haar measure of the hypergroup Ilg^^ is given by 

g-gy r 

f{V^)A{rydr 

with ^ = y-l{q-l)-l = y-^. 

Remark. The specific normalization of the Haar measure is motivated by Theorem 13. Ill below. 

Proof. Ad (a). Clearly Sr 6g is a probability measure on Hg and 

supp((5,. Sg) = {\/r2 + s2 + rvs + sv*v : v G L)g|, 

which does not depend on y. The stated support inclusion is immediate from estimate (IXTUD . 
This shows property (2) in the definition of a hypergroup. (3) is clear because it is known to be 
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true for indices fj, = pdj2 which lead to an orbit hypergroup structure. Property (4) is obvious, 
and (5) with r = r is again true in general because it is true in the orbit hypergroup cases. For 
the proof of (1) it suffices to show that for each / G (^^(ng) the mapping (r, s) i—> /(r*^s) is con¬ 
tinuous. But this is clear from the continuity of the map {r,s,v) -|- rvs + sn*r) 

on Hg X Dq. Commutativity of the convolution is clear, and for the proof of associativity it 
again suffices to consider point measures. So let r,s,t £ Ilg and / G Cbijlq)- Then 

= \ [ [ f{H{r,s,t-,v,w))A{I-v*vY~^A{I-w*w)>"~^dvdw=:I{p) 

JDq J Dq 

with a certain Ilg-valued argument H that is independent of the index ji. Similar, 


{5r <5s) *f, 5t{f) = I'ip) 


with some ;U-independent argument H' instead of H. The integrals I{p) and I'{p) are well- 
defined and holomorphic in {/i G C : 94/i >£»—!}. Further, we know that lip) = I'ip) for all 
p = pd/2 with an integer p ^ 2q. By analytic continuation as in the proof of Theorem Id. 51 fuse 
(Id.91) again) we obtain validity of this relation for all p with y{p > g — 1. 

Ad (b). We have to prove that 


/ /(■s y/r)Air)'^dr = f /(\/r)A(r)'^(ir 


for all / G Cc(ng) and s G Ilg. Equivalently, 



/(\/r)A(r)^ idr. 


v*v)^ ^A(r)^ 1 drdv 


(d.13) 


We know from Section rm that this relation is true if p = pd/2 with an integer p ^ 2q. 
Moreover, the stated result is well known in case d = q = \ for all p in question; actually, in 
these cases is just the convolution of a Bessel-Kingman hypergroup (see Section 3.5 of [BHj ). 
So we may assume that dq ^ 2. Let again IT = {/x G C ; fRp > g — l}- Then fRp — ^ > 0 for all 
p G W, and for fixed / both sides of (ItT^ are well-defined and holomorphic as functions of 
p in IT. We shall again carry out analytic continuation with respect to p, based on Carlson’s 
theorem. We proceed in two steps. 

Step 1. For i? > 0 let Br := {r G Ilg : ||r|| ^ i?}. Choose R > 0 with ^ The 

explicit formula for the convolution shows that there exist constants Sq,Ro > 0 such that 
for / G C'c(ng) with supp/ C and for s G Ilg with ||s|| ^ Sq we have 

fis ^) =0 if ||r|| ^ R. 

Suppose that |/| ^ C on Ilg and notice that A(r) ^ ||t||'^. The left-hand side of (13.131) may 
then be estimated according to 


\LHS\i^C f A(r)^^“?fir- / Ail - v*v)"^-^ dv 
J Bn J Dq 

^ C • • vol(Bfi) • 


(3.14) 


Due to the asymptotics (ESI), p I—> is bounded on IT. Thus by our initial assumption on 
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R we arrive at 

\LHS\ = 0(e5H) as ^ oo in W. 

The right-hand side of (ixni) coincides with the left-hand side for s = 0 and is therefore of (at 
most) the same order as a function of As in the proof of Theorem (ld.5|) . we have to substitute 
^ = {z + 2q)^ to obtain holomorphic functions in z on iHz > — 1 which coincide for all z G No 
and are of order in ^ 0. Carlson’s theorem now implies the assertion under the 

above restrictions on / and s. 

Step 2. Let / G Ccijlq) and s G 11^ be arbitrary. For <5 > 0 define /^(r) := f{6r). We shall 
use that the convolution is homothetic, i.e. 

fsir *), s) = f{Sr 6s). 

Fix constants Sq, Rq as in step 1 and choose <5 > 1 such that supp /5 C and ||||| ^ Sq. 
Then 

:= ^ /(s A(r)'^dr 

^ rig 

As fs and | satisfy the conditions of step 1, we conclude that I{s) = /(O), which finishes the 
proof of part (b). □ 


Our next aim is to determine the dual and the Plancherel measure of the hypergroup 
with fj, > g — 1. For s G Ilg, we define 

^s{r) = (p^{r) := J^{^sr‘^s), r G Rg. 

Notice that 

(Ps{r) = ipr(s) (3.15) 

because depends only on the eigenvalues of its argument. Moreover, we have 

Lemma 3.10. Let g, > g — 1. Then for each s G Ilg, ifg = ips belongs to CbiJLq) with ||v9s||cxd = 
(^s(O) = 1. Ifs G Og, then even G C'o(ng). 

Proof. The hrst assertion is immediate from the Bochner-type integral representation (Hmi) 
for J^. Now suppose s € fig. Then r —> 00 in Ltg implies that ^ 00 and also sr^s —> 00, 
because s is invertible. The second assertion thus follows from the fact that vanishes at 

infinity. □ 


From this Lemma together with the product formula (EH) it is immediate that each (ps with 
s G Ilg belongs to Llg^^. The results of Section mi suggest that these Bessel functions actually 
make up the complete dual: 

Theorem 3.11. (1) The dual space ofUg^^ with g > p — 1 is given by 

Rg,/. = {ips = : s e Rg}. 

(2) The hypergroup Rg^^ is self-dual via the homeomorphism 'L : Rgg^ —> Rg^j, s e-i- (p^. Under 
this identification, the Plancherel measure vr^ on Rg,/^ coincides with the Haar measure Ufj,. 


For the proof of part (1) we need the following 
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Lemma 3.12. The set A := {(fg : s G 11^} is closed in with respect to the topology of 
locally uniform convergence. 

Proof. Let {ipsj)jm be a sequence in A converging to a G ^q,^l locally uniformly. If the sequence 
{sj) C Ilq is bounded, then after passing to a subsequence we may assume that sj —>■ s € Ilg 
as j —> oo. Then —> (ps and therefore a = (ps S A. If else the original sequence (sj) is 
unbounded, then after passing to a subsequence we may assume that Sj —> oo. Thus by Lemma 
nnui Psj{f) = Pr{sj) —> 0 for all r € flq as j ^ oo. This implies a = 0 which contradicts the 
convention 0 ^ Llg^^. □ 


Proof of Theorem Ml . 1 1\. In a first step, we establish that Ilg^^ has subexponential growth in the 
sense of m, i-e. for each compact subset K cHq and each c > 1, the Haar measure satisfies 
= o(c™'). Here iL™' denotes the m-fold convolution power of K, the convolution product 
of subsets A,B of a hypergroup {X, *) being defined hy A* B = UxeA yeB supp(<5x * <5y). Once 
subexponential growth is known, Theorem 2.17 of lYU, will imply that the support of the 
Plancherel measure vr^ coincides with the complete dual Hg^^. For the proof of sub exponential 
growth, it suffices to consider the balls Br = {r G Hg : ||r|| ^ i?}. From the support properties 
of *fj, we see that Bff C BmR- Moreover, 


=C- X{rydr= 0 (i? 297 + 2 n^ = as R^oo. 

Thus for fixed i? > 0, we obtain uj^{Bjf') = and the assertion follows. 

In a second step, we determine vr^. The decisive ingredient will be known results about the 
Hankel transform on a symmetric cone. For s, r G flq, define 

Hf,{s,r) = 

Suppose that p > d{q — 1) + 1. Then according to Theorem XV.4.1. of m, the Hankel 
transform 

U^F{s)-.= ! H^{s,r)F{r)X{r)'^dr 

jQq 

defines an isometric and involutive isomorphism of L‘^{flq, A(r)'^dr). The argumentation of [H] 
and Ell, Section 5 shows that this statement actually extends to all ;U G M with p > ^{q — 1), 
i.e. 7 > —1. Let / G L^(ng,u;^). Then F{r) := /(-v/r) belongs to L^{flq, A{r)"'dr) and a short 
calculation shows that 

f{Ps)=f (ps{r)f{r)dujf,{r) = 2~‘>^^Uf,F{^) Vs G fig. 

J Ilgr 

Moreover, by the isometry of we readily obtain 


\f{<Ps)\‘^du;i,{s) = 


[ \fis)\‘ 

JUa 


dw^{s). 


This shows that the Plancherel measure vr^ associated with is given by 


TT 





g{ips)dujf,{s), 


9 S Cc(ng^^). 


Hence the support of tt^, which we already know to coincide with Hg^^, also coincides with 
the closure of the set {p>s ■ s G Hg} in Hg^^ with respect to the topology of locally uniform 
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convergence. The proof of part (1) is therefore accomplished by Lemma IT12L For part (2), 
it remains to verify that T is a homeomorphism. Continuity and surjectivity are clear. For 
injectivity, suppose ips = ^r- Then in view of (IXT^ we have 5s = 5r and the injectivity of the 
Fourier transform of measures on the hypergroup Fl^^^ implies s = r. To check continuity of 
suppose that ips^ 'Ps locally uniformly. Then 5si —> 5s locally uniformly on Flg^^. Levy’s 
continuity theorem (Thm. 4.2.2. in m) imlies that ^ 5s weakly, and hence Si ^ s. □ 


3.5 The limit case p = g — 1. 

Using Corollary l.‘L7l we see that the convolution =t:^ with p > g — 1 can be written in the 
alternative form 

{Sr*i,5s){f) = —[ f{Vr^ + + rP{y)s + sP{y)*r) ]T(1 - dy. (3.16) 

K/. Jbi 

We shall use this representation to determine the limit of the convolution as ^ J, p — 1, where 
it assumes a degenerate form. As p — 1 = pd /2 with p = 2 q — 1, it is natural to expect that the 
reulting limit convolution coincides with the orbit hypergroup convolution *p_i on Fig derived 
from Mqpq-i. In the following, da denotes the normalized surface measure on the unit sphere 
S = {z e Mi^q : \z\ = 1}. 

The coordinate transform P : Dq of Lemma is assumed to be continuously 

extended to . 


Proposition 3.13. As p [ p — 1, the convolution product 5r 5s converges weakly to the 
probability measure 5r * 5s on Fig given by 


9-1 


{5r*5s)if) = K 




f + rP{y)s + sP{y)*r) i). 

dyi... dyq-i da{yq) 


with a normalization constant k > 0. The product * defines a commutative hypergroup 
structure on Rg which coincides with the orbit hypergroup derived from Mq^ 2 q-i sls in 

Section ixn In particular, 

i5r*5s)if) = {5r*p-i5s)if)= j /(\/r-2 + s2 + + {rds)*)da 

and the additional statements of Theorem and Theorem 13.111 extend to the case p = g — 1. 


Proof. For p > p — 1 consider the probability measure 

Pp ■= cf.Hl - \y\^T~‘^ fB{y)dy 

on Mi^q, where Cp ;= — \y\^Y~^dy and 1 b denotes the characteristic function of the 

ball B. It is easily checked that tends weakly to the normalized surface measure da on S 
as p —> p — 1. Indeed, let / G C{Mi^q) and put F{t) := f{Ty)da{y), r ^ 0. Then with 

[fdpp = ^[ F{T){l-Ty-^T^^-^dT ^F{1) as p^ P -1. 

Jb Cp Jo 

This proves that 5r 5s 5r*5s weakly. It is clear that supp((5r* ds) C supp((5r- *p,5s) for 
p > p — 1 which implies the same support inclusion as in Theorem |3IH (a). In the limit p —> p — 1 
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we further obtain that the Bessel functions 

^Ps{r) = s,reng 

satisfy the product formula 

ips{r)ips{t) = / ips{T)d{Sr*St){T) Vs G rig. 

JUg 

On the other hand, consider the orbit hypergroup Xg-i = (ng,*g_i) derived from Mp^q with 
p = 2q — l. Its dual space consists exactly of the Bessel functions (ps{f) = Jp-i{\rs^r) as above. 
In particular, 

ips{r)ips{t) = / Ps{T)d{5r*e-i5t){r) VsGlIq. 

The injectivity of the Fourier transform on the hypergroup Xg-i now implies that 6 ^ *^-1 dt = 
dr* dt for all r, t. □ 

Remarks. 1. We conjecture that the formulas of Corollary 13.81 permit degenerate extensions 
to successively larger index ranges. As soon as the exponent in one of the iterated integrals 
becomes critical, the corresponding integral over B should be replaced by an integral over S. 
More precisely, we conjecture that within the range 

{p eR: g-^{q-k) - I < p ^ Q -^{q - k-1) - l}, k = q-l,...,l 

the following product formula is valid: 

k 

Jp{r^)Jp{s^) = — [ [ Jp{r‘^+ 3“^+ rP{y)s+sP{yyr)Y\{l-\yj\'^)>^~^+^^'^~^'^ 

■dyi... dyk da{yk+i )... da{yq). 

Also, there should be analogous integral representations for within the above ranges of p. 

2. Further properties of the hypergroups concerning their automorphism groups as 

well as stochastic aspects (such as limit theorems for random walks on matrix cones associated 
with Bessel convolutions) will be the subject of a forthcoming paper joint with M. Voit. 

4. Hypergroups associated with rational Dunkl operators of type B 

In the analysis of the previous sections, one may be interested in questions which depend 
only on the spectra of the matrices from the underlying cone Ilg = ng(F). This amounts to 
considering functions and measures on Ilg which are invariant under unitary conjugation. For 
X G Hq we denote by a{x) = ■ ■ ■ ,Cq) G the set of eigenvalues of x ordered by size, i.e. 

^ ... ^ ^g. The unitary group Uq acts on Ilg via conjugation, (n, r) 1 -^ uru~^, and the orbits 
under this action are parametrized by the set Hg of possible spectra a{r) of matrices r G Ilg, 

Sg = = (6, • • • e ^ ^ ^ 0}. 

We mention that Uq is the maximal compact subgroup of the automorphism group of fig; 
inded the latter coincides with GLg = GLq(¥), acting on Hq via {g,x) gxg*. The set Hg 
is a closed Weyl chamber of the hyperoctahedral group Bq = Sq jk which acts on by 
permutations of the basis vectors and sign changes. In Section |31 we saw that Ilg carries a 
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continuously parametrized family of commutative hypergroup structures with /r ^ — 1 , as 

well as additional orbit hypergroup structures for /i = pd/2, p ^ q an integer. In the following 
section we are going to show that under the above action of Uq on Ilq each convolution =t:^ 
induces a commutative orbit hypergroup convolution on Hg, similar as orbit hypergroups 
were obtained from (commutative) groups in Section f 2 ..S[ In Section lOl we shall then identify 
the characters of (Hg, o^) with multivariable Bessel functions of Dunkl type which are associated 
with the root system of type Bq. In effect, we thus obtain a continuous series of commutative 
hypergroup structures on the chamber Hg whose characters are given by Dunkl-type Bessel 
functions. 

4.1 Convolutions on the spectra of positive definite matrices 

In the situation described above, the canonical mapping 

TT : Ilg ^ Sg , r a(r) 

is continuous, surjective and open with respect to the standard topologies on both sets. There¬ 
fore the map Ilg'^ ^ Hg, Uq.r i—> a(r) becomes a homeomorphism when Ilg'’ is equipped with 
the quotient topology. In the following, du denotes the normalized Haar measure on Uq and 
^ £ Hg is always identified with the diagonal matrix diag(^i, ... ,^g) £ Ilg without mentioning. 
Moreover, we introduce the index set 

T)d 

•= {Y’ p = + !’• • • }u]p - 

Theorem 4.1. (1) For each p £ A4q the chamber Hg carries a commutative hypergroup struc¬ 
ture with convolution 

■= [ if °'^)iC*t^ur]u~^)du, / £ C'(Hg)- 

JUq 

The neutral element of the hypergroup Hg^^ := (Hg,o^) is 0 £ Hg and the involution is given 
by the identity mapping. 

(2) A Haar measure on Hg^^ is given by 

Y = 7 r(w^) = d^h^{i)di with h^{i) = 

2=1 i<j 

and a constant Y > 0. 

Remarks. 1. The constant Y be determined in Section EH 

2. In the generic case p > g — 1 the convolution can be more explicitly written as 

Ofj. dn)if) = ~ I I f{^{V + ^vur]u~^ + upu~^v*^)) A{I — v*vY~^dudv. 

JDq J Uq 

We start with some preparations for the proof of Theorem 14.II 

Lemma 4.2. For u G Uq consider the automorphism of Ilg given by Tu : r uru~^. The image 
measure of the convolution product Sr */j. Sg £ M^ijlq) under T^ is given by 

TuiSr Ss') dy^ru~^ dugu—l. 

Proof. Recall that the measure A(/ — v*v)^~^dv in the convolution formula of Theorem I, ‘1. 91 is 
invariant under unitary conjugations. This yields immediately that 

{g o r„)(r s) = g(uru~^ usu~^) y g G C'(ng). 
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□ 


Aside: this lemma just says that the automorphism of 11 ^ is actually a hypergroup auto¬ 
morphism for each of the convolutions The next lemma is a special case of m, Theorem 
VI.2.3. 


Lemma 4.3. For integrable functions 5 : Ilq —> C, 


'n, 


g{r)dr = K / g{u^u 

Ju, 




with a normalization constant k = Kq > 0. 


Proof of Theorem EH Ad (1). We shall employ the technique of pe], Section 13 for the transfer 
of hypergroup structures via orbital mappings. Notice first that the continuous open surjection 
vr : Bg —> Eg is also proper (because Ug is compact) and thus provides an orbital mapping from 
the hypergroup onto Eg in the sense of [Je], Section 13. For (, G Eg define 

. I 

JUa 


IS 


which is a probability measure on lig and satisfies supp = vr ^ (,^). We claim that each 
vr-consistent in the following sense: 

7 r(eg ds) = 7 r(eg 5t) for all s, t E Ilq with 7 r(s) = vr(t); (4.1) 

here tt is extended to M~^{Jlq) by taking image measures. For the proof of (EH), suppose that 
E Fig satisfy 7 r(s) = 7 r(t). Then for each g E C'(ng) which is invariant under Ug-conjugation 
we have 


/ 


gd{e^*^5s) = / g{uf,u ^*^s)du= / g{f.*f,u '^su)du. 


'Ua 


'Ua 


Notice that for the second identity Lemma [4.2l has been used. As s and t have the same spectra, 
the last integral does not change when s is replaced by t. This proves EH- The orbital mapping 
vr also satishes 7 r( 0 ) = 0 E Hg and 7 r“^( 0 ) = 0 E Fig. We can now apply Qe], Theorem 13.5.A. 
This shows that Eg becomes a commutative hypergroup with convolution 

dg = T(e^ ^r;), 

the identity mapping as involution and neutral element 7 r( 0 ) = 0. This proves the assertions of 
part ( 1 ). 

Ad (2). According to Theorem 13.3.A of [Je], a Haar measure on Hg^^ is given by the image 
measure of under tt. Let / E CciEg) and put g = f o n, which is 17g-invariant. Then 

<? 


f fduj^ = kI gdujf, = k' f 

JEg JUq JSq 


-ijTdi 


where = (\/^, • • •, v^)- ^ constant factor, the last integral coincides with 9 {C)hii{f,)df,. 

□ 

Remarks. 1. Recall that for g = pd/2 with an integer p q the hypergroup Ilg^^ is just the orbit 
hypergroup obtained from the multiplication action of the unitary group Up on Mp^g. In this 
case, the above hypergroup structure of (Hg,o^) can also be described as an orbit hypergroup 
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derived directly from Mpq, as follows: Consider the action of the group L := Up x Uq on Mpq 
by 

X >—>■ uxv~^, {u,v) € L. 

The orbits of this action are parametrized by the possible sets of singular values of matrices 
from Mp^q. Indeed, let agingix) = cj(\/x*x) = (^i ,... ,^q) G Eq denote the singular spectrum 
of X G Mp^q, the singular values being ordered by size. We have the equivalences 


(Tsing{x) = (Tsingiv) Cr{x*x) = (j{y*y) 

G C/g : y*y = vx*xv~^ = {xv~^)*{xv~^) 

^ Up X Uq : y = uxv~^. 

Therefore the orbit space can be identified with the chamber Eq via UpxUq^ (Xsingix), 
and this is easily checked to be a homeomorphism with respect to the natural topologies on 
both spaces. Notice that {uxv~^)*uxv~^ = vx*xv~^. Hence under the mapping <1> : Mp^q —> 
Hg, x i-H’ y/x*x, the above action of L on Mp^q induces the conjugation action of Uq on Hg. 
Moreover, the orbit convolution on Hg^^ is defined in such a way that 


— *^{Qr * Qs) 


with Qr = 


I ^Uaor du ^ M^{Mp^q) 

Ju„ 


where * denotes the usual convolution on the additive group Mp^q. This shows that ioi y = pd/2, 
the convolution of the hypergroup Hg^^ coincides with the convolution of the orbit hypergroup 
^ Hg, which is in turn naturally identified with the convolution of the Gelfand pair 
(L K Mp^q,L). 


2. For y = pd/2 with an integer p ^ q, the support of the measure 6^ 6^ describes the set of 

possible singular spectra of sums x + y made up by matrices x,y £ Mp^g(F) with given singular 
spectra ^ and p. 


Let us return to general indices y G Aiq. In analogy to Lemma l2.bl for orbit hypergroups 
from groups, we expect that the characters of the hypergroup Eg^p are all obtained by taking 
Rg-means of the characters of Hg^^. For ^ G Hg, define £ C'b(Hg) by 

'tp^{p) := / (p^{upu~^)du 
JUq 

where y>^{r) = The are the characters of Hg^^ which are parametrized 

by diagonal matrices. Note that by I/g-invariance of we have 

ips{uru~^) = Pu-isu{r) Vr,s G Hg. (4.2) 

Hence the mean of ps equals the mean of for ^ = a{s) and = p>ri{P,) for all P,,p £ Hg. 

Theorem 4.4. Let y £ Mq. 

(1) The dual space of the hypergroup Eq^p = (Hg, o^) is given by 

~ {^5 = ^ G '=■<?}• 

(2) The hypergroup Eq^p is self-dual via the homeomorphism Eq^p —> Hg^^, P ■ Under 

this identification, the Plancherel measure Tip ofEq^p coincides with the Haar measure cUp. 
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Proof. Ad (1). It is easily checked that each is multiplicative w.r.t. and therefore belongs 
to Indeed, for rj, f ^ Eg we calculate 


/ {'tp^o7r){r]*^uCu ^)d' 
JUq 

By Lemma lO this equals 



)d{6n *fj, 6u(;u-^){r)dudv. 



*n vu(u ^)dudv 


'^dd)'>PdO- 


It remains to show that each character of is of the form with some ^ G Hq. For this, 
notice first that the hypergroup Hq^^ has sub exponential growth, jnst as Ilq^^. Thns according 
to Theorem 2.17 of m, the support of the Plancherel measure vr^ of Hq^^ coincides with the 
fnll dual Hq^^. Let ip G Hq^^ = supp^^. Then by Corollary 6 of m there exists a sequence of 
fnnctions /„ G C'c(Hq) such that fn°fifn converges to pj locally nniformly. Hence 
converges to ^p o tt locally nniformly on Hq, which implies that pj o n is positive definite on the 
hypergronp Hq^^. Notice that (pj o 7r)(0) = 1. Thus by Bochner’s Theorem for commutative 
hypergroups (Thm. 12.3.B of [Jej) and the self-duality of Hq^^, there exists a probability measure 
a G such that 

{p> o TT){r) = / ips{r)da{s) VrGHq. (4.3) 

Jliq 

As i/’ o TT is invariant under the action of Ug and in view of the measure a must be 

t/q-invariant as well, i.e. f f{uru~^)da{r) = f f{r)da{r) for all / G Cc(nq). Let a denote the 
image measure of a under vr. Then with rj = 7r(r) relation dSl) becomes 


= / Ps{r])da{s) = / P’dv)da{p). 
J rio “o 


On the other hand, by Bochner’s Theorem for the hypergroup Sq^^, the character pj of Hq^^ is 
an extremal point of the set of positive definite functions / on the hypergronp Hq^^ with the 
additional property /(O) = 1. This implies that d must be a point measure, i.e. d = 6^ for some 
p G Eg. Hence pj = pj^. 


Ad (2). The self-duality of Eg^^ is proven in the same way as that of Hq^^ fTheorem l3.11|) . To 
determine the Plancherel measure, let / G Cc(Hq) and g := f ott. Then the Fourier transforms 
of / and g w.r.t. the hypergroup structures Hq^^ and Hq^^ are related via 

9{s) = [ gir)Psir)dujg,{r) = f fiP)pJa{s)iP) d(df,iP) = (/o7r)(s). 

J riq J 

By the Plancherel theorem for Hq^^ we readily obtain |/This finishes 

the proof. □ 


We would like to write the characters p!^ in a more explicit form. Recall that (^s(r) = 
i7/j,(jsr^sj for r, s G Hq, where 77^ is given in terms of the spherical series (IT^ . The spherical 
polynomials satisfy the prodnct formula 

Zx(r)Zx(s) f 

--=/ ZxWrusu x/r)du yr,s€Ug, 

JUq 

see ES, Cor. XL3.2. or [UrT], Prop. 5.5. This implies an integral representation for the Bessel 
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functions of two matrix arguments (recall Section f2.21) : 

J^{r,s)= / J^{y/rusu~^y/r)du, r, sElIg. 

JUg 

Thus for r/ E Hg we have 

= ^/.(y^y)- (4.4) 

We shall use this representation in order to identify the characters 'ijj^ with Dunkl type Bessel 
functions for the root system of type Bq. 

4.2 Bessel functions associated with root systems 

Bessel functions associated with root systems are an important ingredient in the theory of 
rational Dunkl operators, which was initiated by C.F. Dunkl in the late 80ies i [D1 j . [D2j i. They 
are a symmetrized version of the Dunkl kernel, which is the analogue of the usual exponential 
function in this theory. As a subclass, they include the spherical functions of a Cartan motion 
group, c.f. |3J2]. In this section we give a brief account on Dunkl theory and the associated 
Bessel functions; for a general background, the reader is referred to [DXj . and m- 

Let G be a hnite reflection group on (equipped with the usual Euclidean scalar product 
(., .)), and let R be the reduced root system of G. We extend the action of G to C? and (., .) 
to a bilinear form on x C^. A funciton k : R ^ C which is invariant under G is called 
a multiplicity function on R. Important special cases of reflection groups are the symmetric 
group Sq which acts on by permuting the standard basis vectors e^, and the hyperoctahedral 
group Bq = Sqt< Z 2 which acts by permutations of the basis vectors and sign changes. The root 
system of Bq is given by i? = {cj, 1 ^ z ^ g} U {±6* ± ej,l ^ i < j ^ q}, and a multiplicity on 
it is of the form k = (ki, ^ 2 ) where ki is the value on the roots Te* and /c 2 is the value on the 
roots ±ej ± ej. 

For a finite reflection group G and a fixed multiplicity function k on its root system, the 
associated (rational) Dunkl operators are defined by 

Tg = mk) =d^ + ^Yl ^ e C''; 

here aa denotes the reflection in the hyperplane perpendicular to a and the action of G is 
extended to functions on via g-f{C) = The are homogeneous of degree —1 on 

the space V = C[C'^] of polynomial functions on and they commute: i |D1 j i. 

Hence the map ^ extends uniquely to a linear map p 1 —> p{T), V —> End(C^). The Dunkl 

operators induce a sesquilinear pairing 

\p,q]k = {p{T)q){0) 

on V, where q{^) '■= q{0- In the following we assume that k is non-negative. Then \p,q]k is 
actually a scalar product on V, see Prop. 2.4. of m- Moreover, for each fixed rc E C*?, the 
joint eigenvalue problem 

m V^eC''; /(0) = 1 

has a unique holomorphic solution f(z) = Ek{z, w) called the Dunkl kernel. It is symmetric in its 
arguments and satisfies Ek{Xz,w) = Ek{z, Xw) for all A E C as well as Ek{gz,w) = Ek{z,gw) 


28 




Bessel convolutions on matrix cones 


for all g £ G. The generalized Bessel function 

Jk{z,w) := -^ ^Ek{z,gw) 

' ' g&G 

is G-invariant in both arguments. Moreover, g{z) = Jk{z, w) is the unique holomorphic solution 
of the ’’Bessel system” 

p{T)g = piw)g Mp E 5 ( 0 ) = 1 (4.5) 

where denotes the subalgebra of G-invariant polynomials in V, see m . For crystallographic 
reflection groups and certain values of k, the operators p{T), when restricted to G-invariant 
functions on constitute the system of invariant differential operators of a Euclidean-type 
symmetric space and the Bessel functions Jk{- ,w) can be identified with the associated spher¬ 
ical functions; for details see |d.T2j . The Dunkl kernel gives rise to an integral transform on 
called the Dunkl transform. Let denote the weight function 

'UJkix) = 

aeR 

on R'^. The Dunkl transform is the integral transform on ,Wk) defined by 

/ ^ P, = f{x)Ek{-ii,x)wk{x)dx (C £ R”^) 

with the constant 

Cfc := [ ^‘^Wk{x)dx. 

A thorough study of this transform is given in m- It has many properties in common with 
the usual Fourier transform to which it reduces in case A; = 0. In particular, the Dunkl trans¬ 
form (as normalized above) extends to an isometric isometric isomorphism of ,Wk), and 

differentiable / of sufficient decay. It is a long-standing open 
question whether L^(R'^,t(;fc) can be given the structure of a commutative Banach algebra so 
that the Dunkl transform becomes the Gelfand transform on its (symmetric) spectrum, sim¬ 
ilar as for commutative hypergroups. In the rank-one case there is such a convolution, but 
it is not positivity-preserving. For details and affirmative results in this direction see [TI,2j . It 
is however conjectured that for arbitrary G and k ^ 0, the Bessel functions Jk have a posi¬ 
tive product formula which leads to a commutative hypergroup structure on a distinguished 
closed Weyl chamber H of G, the dual of this hypergroup being made up by the functions 
^ Jfc(^,?y), rj E H. In rank one and in all Cartan motion group cases this is true, see |R2j . 

In the following, we shall confirm this conjecture for three continuous series of multiplicities 
for root system Bg. Indeed, we shall identify the characters of the hypergroups in Section 
mu with Dunkl-type Bessel functions for Bg and thus obtain hypergroup structures with these 
Bessel functions as characters. 

4.3 Dunkl theory and the convolutions on the Weyl chamber 

Let us denote by the Dunkl-type Bessel function associated with the reflection group G = Bg 
and multiplicity k = {ki, ^ 2 ), and by [., . the associated Dunkl pairing. For z = {zi,..., Zg) £ 
we put z‘^ = {z‘f,...,Zg). The following key result identifies with a generalized oFi- 
hypergeometric function of two arguments (recall the notions of Section f2.21) : 
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Proposition 4.5. Let k = {k\,k2) ^ 0 and k2 > 0. Then for all z,w £ 

lip' 1 1 

= oFi{n-, —) with a = — ,fj. = ki + {m- l)k2 + -■ 

This result was already mentioned in Section 6 of |BFj . but the reasoning there is rather 
sketchy, and there is an erraneous sign in one of the arguments. We therefore include a proof 
by different methods. 


Proof. The modified Jack polynomials px{z) = Cf{z‘^), indexed by partitions A ^ 0, are ho¬ 
mogeneous of degree 2|A| and form a basis of the vector space for G = Bq. Thus the Bessel 
function has a homogeneous expansion of the form 

JkP,w) = ^aA(in)pA(^;) (4.6) 

A^O 

with certain coefficients ax{w) £ C. In view of the Bessel system we have 
P\{TPJk{z,w)\^^^ = px{w)J^{0,w) = 

where the superscript 2 : indicates operation w.r.t. the variable 2 . On the other hand, the results 
of |BFj (relation (2.9) and the formula on top of p. 214) imply that the px are orthogonal with 
respect to [., .]^ with 

\p\,P\\k = 4l^l|A|! (/i)^ • C'^(l) =: Ma, p = ki + {m- l)k 2 + ^ . 
Differentiation of (14.611 (recall that pi, has real coefficients) now gives 

P\{TPJk{z,w)\^^^ = '^ax{w)\px,Pu]k = Mxax{w). 

Hence ax{w) = which implies the assertion. □ 


As a consequence of ( 1231 , Bessel functions associated with a symmetric cone can now be 
identified with Dunkl Bessel functions of type Bq with specific multiplicities: 


Corollary 4.6. Let be an irreducible symmetric cone inside a Euclidean Jordan algebra of 
rank q. Then for r,s £ Q with eigenvalues = {fi, ■ ■ ■ ,fq) and ry = (ryi,..., rjq) respectively, we 
have 




where k is given by k = k{p, d) = — ^{q — 1) — |). 


Now consider again the hypergroup structures = (Hq,o^) on 

Sg = {C = (6, ...,^g)£R‘^:flP...PCgP0} 

which is actually a closed Weyl chamber for the reflection group Bq. The consequence of our 
above identification can be formulated in a twofold way: 

Corollary 4.7. The characters of the hypergroup Eq^^, p £ Aiq are given by 

PriiO = Jki^Ph), h^^g, 
with the multiplicity k = k(p, d) as in the previous corollary. 
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Corollary 4.8. Consider the root system of type Bq with a multiplicity k = (fci,/c2) where 
^2 = f with d € {1,2,4} and ki = |(p — q + 1) — \ for integer p ^ q or arbitrary ki ^ 
^{dq — 1). Then the associated Dunkl-type Bessel functions | characters 

of the hypergroup {Eg, o^) on the closed Weyl chamber Eg, where p = ki + {q — l)k 2 + ^ and 
the convolution is dehned over F = R, C,H, depending on the value of d. In particular, the 
Bessel function satisfies the positive product formula 

JkiC,z)Jk{v,z) = [ J^{C,z)d{S^Of,Sr,){C) V^,?7 G “g, 2: G C'?. 

The hypergroup Fourier transform on is given by 

fiv) = / 

-/Sg 

with = dfj,h^{^)d^ as in Theorem 14.11 Notice that coincides up to a constant factor with 
the weight Wk for k = k{p, d). As Wk is Bg-invariant, we therefore have 

/(r/) = const • F^(? 7 ), 

where F denotes the Bg-invariant extension of / to R'^ and its Dunkl transform. Notice 
that F^ is Bg-invariant as well. With the Plancherel Theorem for the Dunkl transform at hand, 
we are now also in a position to determine the normalization constant d^ of uj^ as announced 
previously. Indeed, recall from Theorem 14.41 that the Plancherel measure of the hypergroup 
Eg^^ coincides with under the natural identification of Eg^^ with its dual. Using this and the 
Plancherel theorem for the Dunkl transform, we readily obtain 

J \ -q 

and 



The value of can be calculated explicitly; it is a particular case of a Selberg type integral 
which was evaluated by Macdonald HD for the classical root systems. 

In the general Dunkl setting, there is a generalized translation on suitable function spaces 
which replaces the usual group addition to some extent, see EH, m and the references cited 
there. On ,Wk), this translation is defined by 

= Cfc' / fHC)EkiiC,C)Ek{ivX)wk{C)dC. 

Jmi 

On has ; L‘^{M.'^,Wk) —> T^(R^,rcfc) with (Tg/)^^(C) = Ek{ir],C,)f^{C). If we restrict to the 
Weyl group invariant case for Bg with multiplicities as in Corollary 14.8L then this generalized 
translation just coincides with the translation defined in terms of hypergroup convolution. If, 
say, / belongs to and is also continuous and Weyl group invariant, then we have 

with the notions of Corollary 14.81 

'TrjfiO = h W) h e ^q- 

When p = ^ with an integer p ^ q, i.e. ki = ^(p — q + I) — then the Bessel func¬ 
tions J^{. , z) can be identified with the the spherical functions of the Cartan motion group 
associated with the Grassmann manifold U{p, q)/(Up x Ug). This follows from the discussion in 
|dJ2| (see also |R.2j i. and is in accordance with Remark o which implies that for p = ^, the 
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hypergroup convolution of coincides with that of biinvariant measures for the Gelfand pair 
{{Up X Uq) X Mp^q, Up X Uq). The multiplicative functions coincide with the (elementary) spher¬ 
ical functions of this Gelfand pair. Thus the hypergroups Eq^p with ^ p — 1 interpolate the 
discrete series of convolution algebras derived from the tangent space analysis on Grassmann 
manifolds. 
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